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ABSTRACT 


This  final  report  of  the  Military  Training  Centers  Project  at  the 
University  of  Illinois  provides  an  administrative  overview  of  the  major 
findings  from  a  five-year  longitudinal  study  of  factors  affecting 
in^lementation  and  operation  of  computer-based  education  in  military  settings* 
Most  of  the  findings  have  since  been  found  generallzable  to  academic  and 
industrial  training  situations  as  well*  An  annotated  list  of  the  24  detailed 
reports  produced  during  the  project  is  included  as  an  appendix* 
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INTRODUCTION 


The  application  of  Computer-Based  Education  (CBE)  provides  innumerable 
traps  for  the  unwary  or  inexperienced*  The  diversity  of  settings,  objectives, 
and  resources  in  which  CBE  exists  makes  application  of  a  few  general 
approaches  to  all  problems  hazardous.  Rather,  methods  which  apply  to  specific 
classes  of  circumstances  oust  be  selected  from  sets  of  alternatives.  These 
alternatives  must  be  drawn  from  experience  in  the  widest  possible  range  of 
settings  in  order  to  avoid  "solutions"  that  are  neither  transferable  nor 
durable. 

This  report  provides  an  overview  of  conclusions  drawn  from  a  program 
designed  to  Identify  such  alternative  approaches  useful  in  implementing  CBE  in 
a  variety  of  settings.  The  program  was  funded  by  the  Department  of  Defense 
through  its  Defense  Advanced  Research  Projects  Agency  (DARPA). 

A  program  of  implementation  of  one  advanced  form  of  CBE,  the  PLATO 
system,  was  begun  by  DARPA  in  the  1960s.  In  order  to  maximally  benefit  from 
the  experience  developed  through  implementation  of  this  new  instructional 
technology,  DARPA  also  provided  for  a  coordinated  support  force,  which, 
beginning  in  1972,  was  to  oversee  and  facilitate  a  series  of  implementations 
in  military  instructional  environments.  One  component  of  this  task  force  was 
the  Military  Training  Centers  (MTC)  group  at  the  University  of  Illinois. 

After  several  years  of  implementations  of  CBE  in  a  variety  of  military 
training  settings,  DARPA  followed  up  with  support  for  a  summative  review  and 
evaluation  of  the  accumulated  experience.  Detailed  findings  from  this  review 
and  evaluation  by  the  MIC  group  have  been  described  in  a  series  of  special 
topical  reports.  Appendix  I  of  this  report  contains  an  annotated  list  of 
these  reports.  In  addition  to  the  military  experience,  the  findings  have 
since  been  tested  and  augmented  by  experience  in  academic  and  Industrial 
settings.  This  final  report  of  the  MTC  project  summarizes  general  findings  in 
a  form  designed  for  quick  review  by  policy  makers.  It  is  based  on  a 
presentation  made  by  the  authors  at  the  1979  Conference  of  the  Association  for 
the  Development  of  Computer-based  Instructional  Systems. 


More  specialized  reports  in  the  MTC  series  should  be  consulted  for 
detailed  descriptions  of  findings.  Specifically,  the  reader  is  directed  to 
Francis  (1977)  for  information  on  establishment  of  a  CBE  site,  to  Call-Hinwich 
and  Steinberg  (1977)  for  information  on  design  of  instruction,  and  to 
Steinberg  (1977)  for  a  study  of  major  factors  affecting  success  and  failure  of 
CBE  site  implementations. 
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PLANNING  THE  PROJECT 

Lynn  Mlsselt 

Control  Data  Education  Company 

As  we  look  at  what  Is  happening  in  the  field  of  C6E  today,  we  see  many 
new  projects  starting  and  CBE  being  applied  in  increasingly  varied  settings. 
CBE  continues  to  be  used  in  schools,  colleges,  and  universities,  but  is  seeing 
increasing  use  In  military  and  Industrial  training.  The  persons  and 
institutions  starting  new  projects  must  either: 

o  Make  decisions  without  experience  (and  be  prepared  to  accept  the 
risks  that  this  implies) 

o  Hire  experienced  people  to  help  make  decisions  (this  option  depends 
on  the  availability  of  experienced  personnel  and  the  applicability 
of  their  experience) 

or 

o  Investigate  a  variety  of  projects  to  become  aware  of  Issues, 
alternatives  to  dealing  with  them,  and  the  trade-offs 
associated  with  the  alternatives. 

As  a  group  we  have  seen  many  new  projects  make  uninformed  decisions. 

We  have  also  seen,  recently,  certain  projects  make  a  conscious  effort  to 
investigate  alternatives  prior  to  making  full  commitments.  We  advocate  the 
latter  approach.  In  this  chapter  I  will  summarize  some  of  our  experience  with 
alternative  approaches.  While  these  approaches  are  generallzable,  they  are 
Intended  to  be  suggestive  and  Illustrative  rather  than  exhaustive. 

The  base  of  experience  we  drew  on  consisted  of  both  direct  experience 
as  CBE  designers  and  developers  and  a  multi-site,  multi-project  evaluation  of 
CBE.  We  began  with  direct  development  experience  as  individuals  or  as  members 
of  development  projects.  Then,  under  support  of  the  Defense  Advanced  Research 
Projects  Agency  (DARPA)  worked  together  as  a  Military  Training  Centers  (MTC) 
group  at  the  Computer-based  Instruction  Research  Laboratory  (CERL)  at  the 


University  of  Illinois,  Urbana-Champaign.  We  attempted  to  train  members  of 
new  projects  in  processes  of  project  planning,  instructional  design,  and 
programming  based  on  our  own  prior  experience. 

We  then  observed  problems  at  individual  sites  and  saw  the  effects  of 
transferring  some  initial  conceptions  regarding  group  structures,  roles,  and 
authoring  models  to  new  settings.  We  also  saw  consequences  of  decisions 
made  (or  not  made)  by  personnel  in  individual  projects. 

After  two  and  a  half  years  of  training,  liaison,  and  support  activities, 
the  MTC  group  shifted  to  an  evaluation  role.  The  purposes  of  this  part 
of  the  project  were:  a)  to  document  experience  gained  at  individual  sites, 
and  b)  to  supplement  the  evaluations  done  by  the  sites  themselves. 

Many  reports  were  generated  over  a  wide  range  of  topics. 

The  support  and  impetus  for  this  evaluation  activity  also  came  from 
DARPA.  DARPA  has  made  a  valuable  contribution  to  the  field  of  CBE  through  its 
support  for  development  of  materials  at  each  site,  and  through  support  of  the 
MTC  group  for  documentation  of  this  experience. 

Finally,  following  conclusion  of  the  DARPA  projects  and  disbandment  of 
the  MTC  group  in  late  1977,  the  authors  of  this  report  have  continued  in  the 
field  of  CBE  and  have  refined  their  insights  as  they  have  applied  them  to  new 
settings — particularly  in  professional  continuing  education  and  in  business 
and  Industry. 

This  report  considers  five  phases  in  the  life  of  CBE  sites — covering 
the  span  from  project  Inception  (birth)  to  maturity.  We  will  report  primarily 
about  groups  responsible  for  all  phases  of  a  project,  from  planning  to 
staffing,  development.  Implementation,  and  final  evaluation  of  courseware. 

Our  major  theme  is  that  persons  responsible  for  new  projects  should  be  aware 
of  the  various  alternatives  open  to  them  in  each  phase  of  the  project.  They 
must  be  aware  of  the  consequences  of  alternatives  and  must  consciously  weigh 
the  alternatives.  This  chapter  deals  with  the  topic  of  project  planning. 

RATIONALE 

There  are  many  reasons  why  a  new  project  ought  to  do  some  good 
planning  and  virtually  no  reasons  for  choosing  to  skip  the  planning  phase. 
Planning  should  be  non-controversial .  Everyone  is  in  favor  of  adequate 
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planning.  Nevertheless,  planning  is  not  always  done  well  or  completely. 

Either  what  is  involved  in  planning  is  not  understood  (i.e.,  the  required 
products  of  the  planning  phase  are  not  apparent),  or  planners  are  not  aware  of 
all  the  alternatives,  trade-offs,  and  pitfalls  that  must  be  considered.  I 
shall  outline  some  of  the  most  basic  issues  which  must  be  dealt  with  in  the 
planning  phase. 

Project  planning  is  an  extremely  broad  topic  in  that  it  is  an  activity 
that  should  continue  throughout  the  life  of  the  project.  The  topic  of 
planning  overlaps  with  the  other  chapters  in  this  report. 

Since  there  are  literally  hundreds  of  issues  that  should  be  addressed 
in  planning  and  managing  a  project,  I  cannot  be  exhaustive.  However,  I  hope 
to  illustrate  the  process. 

In  considering  how  to  approach  the  topic  I  have  had  to  choose  between 
either  describing  the  planning  done  in  various  sites  studies  and  the  pitfalls 
encountered  in  each  or  suggesting  a  general  approach  to  be  taken  by  planners 
of  new  projects.  I  have  decided  to  do  the  latter. 

The  planning  phase  should  address  the  major  management,  issues  to  be 
considered  in  each  of  the  remaining  phases.  Problems  or  decisions  that  are 
not  anticipated  can  be  the  total  undoing  of  a  project. 

The  products  of  the  planning  phase  should  Include  the  following 
documents : 

o  A  statement  of  project  goals  together  with  lists  of  assumptions 
and  constraints  that  have  shaped  the  goals, 
o  A  plan  for  staffing  and  assigning  staff  to  tasks, 
o  A  plan  for  arranging  facilities  and  hardware, 
o  A  plan  for  implementing  the  courseware  in  the  Instructional 
environment — i.e.,  a  plan  for  assuring  usage, 
o  A  plan  for  final  evaluation  and  follow-on  activities. 


These  plans  should  represent  a  thorough  analysis  of  the  Issues 
relevant  to  each  phase.  They  should  show  evidence  of  consideration  given  to 
alternatives.  The  plans  should  be  flexible;  they  should  be  based  on  what  is 


known  vor  assunnil ,  but  thuy  sliould  be  capable  of  being  modified  in  light  of 
changing  circunistances. 

DOCb’MENT  GOALS  Af!D  CONSTRAINTS 

The  first  step  in  the  planning  phase  should  be  the  process  of  goal 
clarification.  There  are  many  potential  reasons  for  becoming  involved  with 
CBE.  These  include:  a)  Improvement  of  on-the-job  performance;  reducing  costs 
through  reducing  errors,  b)  reduction  of  costs  of  training  by  shortening 
training  time  or  eliminating  expensive  equipment,  c)  improvement  of 
motivation,  thus  reducing  attrition,  d)  standardization  of  training, 
e)  presentation  of  tests,  prescription  of  learning  activities,  and  record 
keeping — functions  handled  by  computer-managed  instruction,  f)  research  on 
such  topics  as  human  learning,  implementation  of  technology,  methods  of  multi¬ 
person/multi-site  research  and  development,  g)  enrichment  of  the  curriculum  by 
preserving  expensively  taught  courses  and  teaching  topics  that  could  not 
otherwise  be  taught,  and  h)  remediation  in  basic  skills  to  free  the  teacher  to 
deal  with  individual  learning  needs. 

Although  each  of  these  aims  may  be  attainable  in  different  settings  by 
proper  application  of  CBE,  they  are  not  all  mutually  compatible  within  a 
single  project.  There  is  potential  for  conflict  between  research  aims  and 
operational  training,  as  demonstrated  by  the  first  phase  of  the  Chanute 
project.  There  may  be,  for  example,  potential  conflict  between  the  goals  of  a 
narrowly  defined  pilot  project  and  the  goal  of  high  visibility  and  continued 
use  across  many  departments. 

During  goal  clarification,  planners  must  determine  those  goals  which 
grow  out  of  real  needs  and  which  have  the  most  support  in  the  organization. 
After  clarifying  the  goals  for  the  project,  the  project  planners  should 
document  them  together  with  constraints  and  assumptions.  Finally,  the 
planners  should  be  prepared  to  continue  the  process  of  goal  clarification 
throughout  the  life  of  the  project.  There  may  be  a  new  dean  or  new 
superintendent  or  new  general  in  a  decision-making  role  who  does  not 
understand  or  does  not  agree  with  the  established  goals,  or  original 
assumptions  might  prove  invalid  and  the  relative  priorities  may  change.  Once 


the  project's  goals  have  been  clarified,  the  planner  can  move  on  to  the 
problem  of  planning  the  staff  required* 

PLAN  FOR  STAFFING 

The  first  step  is  to  determine  tasks  that  must  be  carried  out  by  the 
project  staff.  A  major  issue  is  whether  the  staff  will  do  development.  The 
alternatives  are:  assemble  a  staff  to  develop  your  own  courseware,  use  the 
courseware  developed  by  others,  or  contract  an  external  vendor  to  do  custom 
development.  Planners  must  consider  the  consequences  of  each  alternative. 

The  planners  must  consider  those  tasks  to  be  done  by  the  project  staff  and 
those  that  can  be  done  by  others  in  the  institution.  The  planners  must  also 
identify  all  the  tasks;  it  is  extremely  easy  to  let  some  slip  through  the 
cracks.  Next  the  planners  must  Identify  the  skills  required  among  staff 
members  to  cover  those  tasks.  Finally,  the  planners  must  obtain  a  staff  with 
the  necessary  skills  either  by  training  individuals  in  the  desired  skills  or 
by  hiring  experienced  personnel. 

PLAN  FOR  THE  MANAGEMENT  OF  THE  DEVELOPMENT  PROCESS 

During  the  planning  phase  the  project  manager  should  establish  a  plan 
for  management  of  the  development  process.  A  first  step  is  to  plan  for  the 
integration  of  CBE  activities  into  a  total  course.  A  total  course  design  is 
based  on  the  training  problem  whose  solution  is  sought,  accounts  for  various 
constraints,  and  provides  an  overall  structure  in  which  individual  activities 
fit.  Planners  cannot  expect  to  develop  individual  lessons  and  have  them  used 
unless  they  are  designed  to  fit  in  a  context.  It  is  also  necessary  to: 

o  Establish  milestones  with  which  to  gauge  progress  during 
development . 

o  Plan  to  collect  data  on  time  being  spent  on  various  activities  and 
compare  these  with  the  plan  and  the  milestones.  Either  adjust  the 
procedures  used  during  development,  adjust  the  staff  assigned,  or 
adjust  the  milestones  and  ultimate  goals. 

o  Plan  the  role  that  formative  evaluation  will  take,  including  reviews 
by  peers,  subject  matter  experts,  student  testing,  and  revision  cycles 
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IMPLEMENTATION 

The  next  major  document  of  the  planning  phase  should  be  a  plan  for 
implementation  and  delivery  of  courseware.  Organizational  acceptance  and 
commitment  does  not  just  happen — you  must  plan  for  it.  The  roles  of 
instructors  and  administrators  vis  k  vis  the  new  courseware  should  be  worked 
out.  Planners  must  be  certain  to  cover  a  real  need  that  can  build 
acceptability  into  the  product.  Resources  should  be  based  on  need —  too 
little  or  too  much  should  be  avoided. 

FINAL  EVALUATION  AND  FOLLOL’-ON  ACTIVITIES 

A  final  planning  document  should  provide  for  final  evaluation  and 
follow-on  activities.  The  evaluation  plan  should  be  oriented  toward  decision 
making.  The  plan  should  Identify  audiences  for  evaluation  as  those  persons 
who  will  be  making  decisions  about  the  courseware  and  the  development  project 
The  plan  should  identify  the  Information  needed  to  make  decisions  while 
planning  for  as  much  redundancy  of  measures  as  can  be  afforded. 

Because  decisions  are  often  made  before  the  final  report  is  available 
the  evaluation  data  collection  and  reporting  should  be  continuous  throughout 
the  project.  Decisions  made  prior  to  project  completion  will  affect  staffing 
and  schedules  during  the  last  phases  of  the  project.  The  plan  must  also 
provide  for  maintenance  and  revision  of  the  courseware. 

As  a  given  project  comes  to  a  close,  the  plan  for  that  project  ought 
to  take  follow-on  activity  into  account.  The  optimum  organization,  staffing, 
and  facilities  for  one  project  may  be  totally  Inappropriate  for  subsequent 
work.  In  addition,  unless  the  prospect  for  follow-on  activities  is  clear  and 
appealing,  staff  members  may  leave  early,  thus  jeopardizing  completion  of  the 
original  project.  If  no  follow-on  activities  are  expected,  the  project 
planner  must  provide  for  an  orderly  shut  down  or  a  maintenance  effort. 

These  have  been  some  of  the  requirements  for  outcomes  of  the  planning 
phase.  Once  these  plans  have  been  assembled  they  should  be  reviewed  by 
project  planners  and  sponsors  In  the  light  of  experience  of  other  projects. 
Gaps  and  potential  pitfalls  should  be  identified  and  provided  for. 


CONTINGENCY  PLANNING 


Planners  must  build  flexibility  Into  their  plans*  It  is  a  fact  of 
life  that  circumstances  change,  assumptions  are  invalidated,  and  new 
constraints  arise,  ^^lrphy's  Law —  Whatever  can  go  wrong  will  go  wrong — 
applies  just  as  well  to  CBE  development  efforts  as  to  other  enterprises.  The 
CBE  manager's  only  defense  against  these  changes  is  to  anticipate  them  and  to 
have  a  second,  back-up  plan  in  the  wings:  a  contingency  plan. 

The  task  of  the  manager  in  contingency  planning  is  simply  to  go 
through  the  entire  set  of  planning  documents  and  ask  the  question  "What  if..." 
For  example,  "What  if  there  is  a  change  in  upper  level  management.  Will  these 
goals  still  be  acceptable?"  "What  if  I  am  not  able  to  hire  and  train  all  the 
people  I  need?"  "What  if  my  prime  subject  matter  expert  is  killed  in  an 
accident?"  "What  if  the  production  rate  is  slower  than  planned?"  "What  if 
the  student  sample  is  thinned  due  to  attrition?  Will  there  still  be  enough 
data  to  answer  the  evaluation  questions?" 

Acceptable  alternatives  to  these  contingencies  should  be  adopted  as 
part  of  the  overall  plan.  If  there  is  not  enough  flexibility  in  the 
constraints  on  the  project  to  allow  for  one  or  more  of  these  disasters,  the 
manager  is  riding  a  time  bomb  that  will  eventually  explode. 

SUMMARY 

This  chapter  has  presented  a  very  general  overview  of  some  of  the 
principles  of  project  planning. 

The  products  of  the  planning  phase  should  include  documents  which 
clarify  the  project's  goals  and  stated  assumptions,  list  the  tasks  for  which 
staff  skills  should  be  acquired,  state  a  plan  for  implementation  and  delivery 
of  the  courseware,  and  provide  for  the  final  evaluation  and  follow-on 
activity. 

Once  the  plans  are  established,  they  should  be  reviewed  in  light  of 
experience  with  pitfalls  in  other  projects.  The  entire  set  of  plans  should  be 
backed  up  with  contingency  plans  that  are  acceptable  to  the  sponsors. 

While  these  guidelines  may  seem  totally  obvious  and  self-evident,  they 
are  not  always  followed.  Detailed  analyses  of  the  planning  done  for 
individual  projects  reveal  examples  of  both  good  and  bad  planning.  We  hope 
that  all  future  projects  will  be  examples  of  good  planning. 
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STAFF  SELECTION  AND  RETENTION 

Larry  Francis 

Battelle  Memorial  Institute 

When  computer-based  education  (CBE)  centers  or  projects  are  described, 
often  the  first  characteristics  listed  are  the  computer  hardware,  terminals, 
and  software.  Next,  perhaps,  the  subject  matter  areas  are  used  to  categorize 
the  activity  at  a  center.  Information  about  the  composition  of  the  staff  is 
typically  given  less  attention;  nevertheless.  Investigations  have  shown  that 
staff  characteristics  and  staff  structure  have  often  been  the  critical 
features  of  a  CBE  center,  outweighing  hardware  and  software  in  terms  of  what 
was  potentially  and  actually  achieved. 

ROLES  VS.  SKILLS 

In  most  fields,  when  considering  what  staff  to  hire,  an  administrator 
examines  the  program  objectives  and  begins  writing  job  descriptions  for  the 
roles  of  the  staff  needed  to  accomplish  the  objectives.  In  the  field  of 
computer-based  education,  this  approach  is  not  always  the  most  practical. 

Because  of  the  limited  availability  of  people  with  CBE-related  experience,  we 
suggest  that  staff  members  be  selected  and  described  on  the  basis  of  skills, 
rather  than  by  roles  (Francis,  1977).  Thus,  instead  of  referring  to  authors,  one 
refers  to  staff  members  with  programming  skills,  subject-matter  expertise, 
and/or  instructional  design  skills.  Because  job  candidates  are  likely  to 
possess  a  crazy-quilt  pattern  of  skills,  premature  combining  of  skills  into 
roles  and  job  descriptions  imposes  unnecessary  constraints  on  the  staff 
selection  task.  Skills  we  have  found  necessary  in  most  CBE  development 
programs  include  (Steinberg,  1977): 

o  Administration/Management 
o  Leadership 

o  Curriculum  coordination 
o  Subject-matter  expertise 
o  Instructor  experience 
o  Instructional  design 
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o  Evaluation/Testing 
o  Prog ramming /CBE  expertise 
o  Editorial  skills 
o  Artistic/Illustrative 
o  Proctoring  skills 
o  Secretarial/Clerical 
o  Experience  with  other  media 

There  are  several  additional  advantages  for  choosing  staff  by  skills 
rather  than  by  roles.  One  of  the  most  consistent  findings  of  a  study  of  PLATO 
courseware  development  procedures  in  military  and  civilian  environments  was 
that  new  courseware  development  groups  changed  their  organizational  structure 
one  or  more  times  during  the  three-year  period  examined  (Mahler,  Mlsselt, 
Schell,  &  Alderman,  1976).  The  upset  caused  by  reorganization  can  be  somewhat 
minimized  by  emphasizing  skills  (which  change  slowly),  rather  than  roles 
(which  are  changed  suddenly  by  the  reorganization).  For  example,  consider  a 
staff  member  who  would  traditionally  be  classified  as  an  author.  If  he  had 
difficulty  mastering  the  CBE  language,  the  administrator  might  assign  him  only 
subject  matter  tasks  and  responsibilities.  Thus  the  staff  member  would  not 
waste  time  or  feel  frustrated  by  trying  to  become  a  well-rounded  author,  but 
would  use  the  skill (s)  in  which  he  is  strong. 

Another  advantage  for  using  a  "skill"  orientation  is  that  such  an 
approach  may  convince  an  overworked  CBE  director  who  has  managerial, 
leadership,  and  curriculum  coordination  functions  that  he  can  continue  in  a 
leadership  capacity,  but  delegate  other  functions  to  subordinates. 

team  authoring 

It  has  been  assumed  in  the  previous  discussion  that  some  sort  of 
"team"  is  being  used  to  develop  CBE  courseware.  Although  any  multi-person 
effort  might  be  said  to  employ  a  team  approach,  it  is  useful  to  distinguish  at 
least  3  levels  of  teams. 

o  Independent  authors  may  divide  the  content,  but  do 
all  parts  of  courseware  development  themselves. 


o  One  staff  member  may  make  the  decisions  about  design,  content,  and 
programming,  hut  "sub-contract"  support  staff  to  do  portions  of 
these  tasks. 

o  No  single  person  may  make  all  the  decisions  about  design,  content, 
and  programming.  In  this  situation  responsibility  for  a  piece  of 
courseware  is  shared  among  the  developers.  Development  is  like  an 
assembly  line  and  control  passes  on  when  a  stage  of  development  is 
complete . 

Advocates  of  this  last  form  of  team  authoring  often  feel  that  they 
have  found  "truth"  and  "THE  method"  in  comparison  to  authoring  courseware  more 
independently.  In  fact,  the  objectives,  time  constraints,  and  author 
backgrounds  strongly  affect  what  solution  is  best  for  a  given  situation. 

Many  courseware  development  groups  start  out  in  the  "team"  mode 
described  first  and  gravitate  toward  approaches  with  a  higher  degree  of 
specialization.  Often  such  a  pattern  is  predictable.  A  loose,  independent 
organization  is  simple  to  set  up  and  operate.  However,  because  few  staff 
members  possess  all  the  skills  needed  to  be  effective  authors,  uneven, 
inconsistent  courseware  begins  to  emerge.  Similarly,  staff  members'  individual 
strengths  and  weaknesses  are  found.  In  order  to  enhance  productivity  and 
increase  uniformity,  the  manager  of  courseware  development  groups  then  decides 
to  assume  the  larger  administrative  burden  associated  with  the  second  and 
third  forms  of  the  team  approach.  Thus,  the  staff's  strengths  are  exploited 
and  the  manager  achieves  greater  control. 

SELECTION  OF  THE  DIRECTOR 

It  is  obvious  that  the  selection  of  a  group  leader  or  project  director 
has  a  major  impact  on  the  success  of  any  endeavor.  Because  qualified  managers 
are  difficult  to  find,  the  management  staff  is  often  over-extended  in  an 
attempt  to  get  maximum  use  from  those  managers  who  are  available.  In  some 
cases  a  manager  will  try  to  oversee  five  or  six  different  programs  or 
projects.  Our  experience  shows  that  whenever  the  staff  members  to  be 
supervised  exceeds  three  full-time  equivalents  or  five  people,  a  full-time 
administrator  is  generally  needed.  We  have  found  two  other  criteria 
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to  be  useful  In  determining  the  appropriate  "load"  for  an  administrator: 

o  The  manager  should  have  enough  time  available  to  see  students 
using  CBE>  Unless  this  time  is  made  available,  the  manager 
tends  to  evaluate  the  CBE  lessons  and  project  only  from  his 
perspective  rather  than  as  the  students/trainees  see  them* 

o  The  manager  should  be  the  appropriate  "distance"  from  the  CBE 
activity:  neither  too  close  nor  too  far.  Some  managers  who 
have  acted  as  both  a  CBE  developer  and  also  as  the  manager  found 
it  difficult  to  focus  on  the  long-term  administrative  issues 
because  the  short-term  development  issues  got  in  their  way. 

At  the  other  extreme,  one  supervisor  found  that  because  of  his 
unfamiliarity  with  the  details  of  his  CBE  project,  he  was  left 
out  of  the  decision-making  process.  His  subordinates  found  that 
he  could  not  quickly  grasp  the  issues  and  hence  made  the  decisions 
themselves. 

SELECTING  CBE  DEVELOPERS 

The  CBE  developers  one  can  hire  typically  have  one  of  three 
backgrounds:  subject  matter  expertise,  instructor  experience,  or  a 
programming  background.  Each  type  of  person  may  become  a  full-fledged  CBE 
author,  but  each  has  a  potential  weakness  or  difficulty  of  which  to  be  aware. 
Subject  matter  experts  may  produce  lessons  which  tell,  but  do  not  teach.  The 
lessons  may  be  technically  sound  but  lacking  interaction,  questions,  and 
pedagogy.  As  a  result,  the  materials  are  boring.  Instructors  who  become 
authors  have  a  better  chance  of  understanding  what  educational  strategies  will 
work  with  students;  unfortunately,  robbing  the  classrooms  to  develop  CBE  may 
be  "expensive"  in  that  classroom  students  must  suffer  from  inexperienced 
instructors.  Former  programmers  tend  to  view  CBE  lessons  without  a  sense  of 
esthetics.  If  a  lesson  executes  properly,  they  may  be  completely  satisfied. 
Students  using  such  a  lesson  may  find  displays  awkward,  introductions  missing, 
and  learning  difficult.  All  of  these  types  of  people  have  been  successfully 
trained  as  CBE  developers;  the  purpose  here  is  to  suggest  potential  areas  of 
weakness  which  demand  the  manager's  attention  or  supplementary  training. 
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fractional  appointments 

The  subject  of  fractional  appointments  can  be  condensed  to  one 
sentence,  "In  CBE  projects  to  date,  there  have  been  too  many  full-time  authors 
and  too  few  full-time  directors".  Directors  have  already  been  discussed; 
part-time  authors  will  be  treated  here.  When  "drafting"  CBE  developers  from 
the  classrooms  for  a  CBE  endeavor,  it  is  often  effective  to  let  them  retain  a 
fractional  commitment  to  their  former  position.  Such  a  staff  member  can 
perform  useful  liaison  activities:  reassuring  current  instructors,  finding 
student  "guinea  pigs",  squelching  rumors,  and  basically  maintaining  rapport 
between  the  CBE  developers  and  the  classroom  staff. 

One  major  problem  with  split  appointments  is  that  actual  CBE  development 
time  tends  to  be  less  Chan  that  officially  scheduled.  Because  test  writing, 
meetings,  grading,  and  other  activities  have  firmer  deadlines  and  are  more 
structured  activities  than  CBE  tasks,  the  CBE  work  suffers.  Separate  offices 
and  definite  time  slots  for  each  kind  of  activity  are  an  aid.  When  a  large 
curriculum  is  being  rapidly  developed,  the  need  for  coordination  among 
developers  is  so  large  that  staff  members  working  less  than  half  time 
generally  find  too  little  time  left  over  for  efficient  work.  However,  small 
fractions  may  be  fine  for  staff  members  not  involved  in  coordination  meetings 
(e.g.,  subject  matter  experts  or  programmers).  Part-time  student  programmers 
have  often  been  employed  with  high  cost-effectiveness. 

STAFF  MORALE  AND  MOTIVATION 

Once  a  staff  has  been  selected  and  trained,  the  administrator's  next 
worry  is  attrition.  Loss  of  trained  staff  members  from  a  CBE  group  can  be 
expensive,  not  only  in  terms  of  replacement  cost,  but  also  in  terms  of  lost 
time  and  unplanned,  but  necessary,  staff  reorganizations.  Because  financial 
rewards  for  a  superior  performance  can  generally  be  given  only  once  a  year  and 
are  subject  to  many  other  constraints,  smaller  non-f inancial  rewards  can  be 
used  to  thank,  motivate,  and  reinforce  a  deserving  staff  member.  Morale  of 
staff  members  is  enhanced  when  the  following  actions  are  taken: 

o  Recognition  of  the  contribution  of  a  staff  member  on  the  credits 

page  of  a  CBE  lesson  or  in  reports,  journal  articles,  proposals,  etc. 
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o  Use  of  a  staff  member's  CBE  lessons  during  a  demonstration.  An 
even  greater  compliment  is  to  have  the  staff  member  give  all  or 
a  part  of  a  demonstration. 


o  Releasing  resources  such  as  disk  space,  computer  time,  and  computer 
access  privileges  for  "side"  projects  or  personal  use  by  staff 
members  or  their  families. 

o  Allowing  individuals  flexibility  in  their  working  schedules.  In 
fact,  because  flexibility  is  potentially  possible  in  most  CBE 
environments  and  is  obvious  to  all,  failure  to  offer  it  may 
cause  discontent. 


o  Giving  an  "ego  boost"  by  scheduling  student  use  of  a  well-tested 
lesson  written  by  an  author.  Students  and  instructors  generally 
make  at  least  a  few  complimentary  statements,  and  the  dwj^ndling  number 
of  corrections  needed  serve  to  remind  the  author  of  just  how  polished 
the  lesson  is  becoming. 
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LESSON  DEVELOPMENT 

Eileen  Call-Hinwich 

Department  of  Research  in  Health  Education 
University  of  Connecticut  Health  Center 

An  author  at  a  remote  PLATO  site  had  just  returned  home  after  a  one- 
week  course  in  coding  which  included  a  smattering  of  instructional  design. 

The  course  was  his  first  exposure  not  only  to  coding  and  instructional  design, 
but  to  computers  of  any  sort  as  well.  He  felt  challenged  by  and  enthusiastic 
about  the  potential  of  CBE.  He  was  eager  to  begin  writing  his  own  lessons. 

He  thought  writing  lessons  was  tremendously  creative  and  he  regarded  himself 
as  a  creative  person. 

During  the  short  training  course,  the  instructor  had  stressed  the 
importance  of  preplanning  as  the  first  step  in  lesson  writing,  with  particular 
emphasis  given  to  the  careful  construction  of  a  list  of  the  tasks  the  author 
wanted  the  student  to  be  able  to  perform  by  the  lesson's  end.  During  the 
training  course,  the  author  had  seen  the  importance  of  such  prior  analysis, 
but  once  he  got  home  and  began  writing  his  first  lesson,  he  found  prior 
planning  rather  tedious  —  a  little  like  having  to  practice  scales  when  what 
you'd  really  like  to  be  able  to  play  is  Beethoven's  "Pathetique. ’’  He  tried  to 
decide  what  material  should  be  Included,  and  how  it  should  be  organized,  but 
he  felt  stifled,  anxious  to  really  "get  down  to  writing  the  lesson."  He 
finally  announced  that  he  was  "too  creative"  to  function  this  way.  His 
"technique"  would  be  to  sit  at  the  terminal  and  compose,  impron5)tu  style, 
adllbbing  as  he  went  along.  This  procedure  felt  much  more  comfortable.  He 
found  himself  bursting  with  ideas.  He  had  seen  some  fascinating  games,  and 
wanted  to  write  a  game;  he  had  seen  some  dazzling  displays,  and  wanted  to 
create  some  graphics.  He  struggled,  but  felt  invigorated  by  the  struggle.  He 
labored,  but  felt  challenged  by  the  labor.  He  rhapsodized,  "Writing  lessons 
is  so  creative  and  personal  —  it's  like  giving  birth  1" 

Slightly  over  a  year  later,  he  bore  a  rambling,  hopscotching, 
patchwork  of  a  lesson.  Throughout  the  year,  most  of  his  time  had  been  eaten 
away  by  reworking,  reshuffling  and  re-wrlting  what  he'd  already  done.  New 
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material  was  tacked  on  to  the  end  of  the  lesson  in  a  rather  "Oh-and-I-f orgot- 
to-mention. . . "  manner.  He'd  constructed  a  game,  but  the  rules  were  so 
complicated,  students  got  too  befuddled  to  play.  He  had  included  some 
graphics,  but  they  were  juvenile  and  cartoonish.  Students  came  to  refer  to 
the  lesson  as  "Sesame  Street." 

That's  a  useful  anecdote  (which,  by  the  way,  is  true)  because  it 
encompasses  not  only  a  number  of  common  design  problems,  but  a  number  of 
managerial  problems  as  well.  Before  discussing  specific  design  issues,  let's 
look  at  the  ways  in  which  misguided  management  decisions  (and  lack  of 
decisions)  helped  undermine  this  situation. 

This  project  was  staffed  by  a  number  of  subject-matter  specialists 
representing  a  wide  range  of  programming  and  teaching  experience.  No  one  was 
familiar  with  with  the  TUTOR  language  or  PLATO  capabilities.  However,  author 
training  time  was  Included  in  production  time.  Time  and  budget  being  primary 
Issues,  authors  were  sent  first  to  a  very  short  course,  and  then  expected  to 
begin  producing  curriculum  materials  immediately.  In  this  way,  each  author 
was  essentially  behind  from  the  very  beginning.  In  addition,  the  time  press 
helped  foster  the  feeling  among  administrators  and  authors  alike  that  the  only 
really  productive  work  —  or  as  our  author  put  it  "getting  down  to  writing  the 
lesson"  —  was  actual  lesson  programming.  The  result  was  a  lack  of  any 
standardized  development  procedure.  Each  author  was  left  to  develop  materials 
in  whatever  way  he  saw  fit. 

Without  dampening  any  "creative"  inclination,  an  alternative  approach 
would  have  been  to  adopt  a  set  of  project  standards  to  which  lessons  should 
conform.  Another  technique  which  has  proved  useful  at  a  number  of  sites  is 
the  development  of  a  driver  which  could  be  used  as  a  mainframe  for  project 
lessons.  Such  drivers  not  only  standardize  some  educational  practices  (such 
as  use  of  student  objectives  and  lesson  "roadmaps"),  but  also  provide 
Individual  authors  with  a  pattern  for  organizing  material  and  routing  students 
through  their  lessons.  Far  from  hampering  creativity,  standardized  drivers 
can  free  authors'  time  to  deal  with  lesson  material  at  a  more  innovative 
level. 

Finally,  since  no  one  on  staff  had  any  knowledge  or  experience  with 
TUTOR  or  CBE,  even  novice  authors  were  left  with  very  little  programming  or 


design  support.  Consequently,  some  lessons  remained  limited  to  heavily  texted 
displays  and  student  interaction  that  was  easiest  both  to  compose  and  code. 

As  for  the  author  in  the  anecdote,  he  fell  prey  to  a  number  of 
pitfalls  that  could  fairly  easily  have  been  averted.  His  biggest  problem  was 
in  not  knowing  how  to  think  about  developing  his  lesson.  Basically,  his 
thinking  centered  on  the  content  rather  than  the  student.  This  orientation  is 
partially  responsible  for  his  continuing  to  tack  added  material  onto  the  end 
of  the  lesson.  An  alternative  (and  ultimately  time-saving  approach)  would 
have  been  to  first  enumerate  the  tasks  he  expected  the  student  to  be  able  to 
perform  and  then  divide  those  tasks  into  their  component  sub-tasks.  Tedious, 
perhaps,  but  in  doing  so  the  author  provides  hlm/herself  with  not  only  an 
interlocking  hierarchical  framework  for  the  lesson  material,  but  also  a  point 
of  reference  by  which  to  weigh  the  relevance  of  both  content  information  and 
possible  preseatation  techniques. 

Another  pitfall  was  visible  in  this  author's  use  of  various  graphics 
and  games  for  their  own  sake  rather  than  for  their  educational  value  or 
relevance.  Without  a  clear  definition  of  the  student  goals,  the  author 
developed  a  somewhat  short-sighted  perspective  toward  presentation.  Graphics 
anfl  games  became  an  end  in  themselves. 

COMMON  DEVELOPMENT  MYTHS 

Perhaps  this  author's  greatest  oversight  was  in  believing  a  number  of 
"myths"  of  lesson  development.  First,  and  doubtless  most  disastrous,  was  the 
myth  that; 

An  author  can  improvise  a  lesson.  Our  author  was  well-intentioned, 
but  misguided  (and  reinforced  by  administrative  attitude)  in  believing  that 
planning  and  writing  a  lesson  are  somehow  separate.  That's  like  saying 
playing  the  flute  and  learning  what  finger  positions  produce  what  notes  are 
two  different  things. 

No  one,  not  even  experienced  authors,  can  simply  extemporize.  While 
it's  true  that  authors  often  "compose"  at  the  terminal,  they  are  able  to  only 
because  they  have  a  firm  basis  of  experience.  This  author  was  misled  by  the 


19 


outward  behavior  of  more  experienced  authors.  He  didn't  see  the  mental 

organization  that  had  gone  on  or  the  fact  that  they  were  able  to  "adlib"  only 

because,  having  written  a  number  of  lessons,  they  had  practiced  and  become 
proficient  at  the  necessary  skills.  Which  brings  us  to  his  second 
misconception: 

If  you  know  how  to  teach,  you  know  CBE.  Any  craft  is  grounded  in  a 
certain  set  of  skills/tools  with  which  the  craftsperson  must  be  proficient. 
They  are  the  heart  of  the  craft.  Just  as  one  would  not  assume  a  pianist  could 
automatically  play  the  flugel  horn,  one  should  not  assume  an  Instructor 
automatically  knows  CBE.  They're  in  the  same  field,  but  require  different 

skills,  approaches,  etc.  They  have  different  potentials  and  capabilities  and 

only  some  similarities. 

Presenting  is  teaching.  The  author  also  believed  that  simply  by 
stringing  some  bits  of  information  together,  he  was  teaching.  A  more  extreme 
example  is  the  medical  CBE  site  at  which  the  emphasis  on  content  alone 
resulted  in  lessons  evolving  into  "page-turners"  —  heavily  textual,  bookish 
lessons  with  little  student  Interaction.  Students  were  strongly  encouraged  to 
sit  at  their  terminals  and  take  notes.  Most  did  —  like  modern  day  scribes, 
conscientiously  transcribing  books  of  their  own.  Expensive  books,  to  say  the 
least . 

Lesson  development  is  a  one-shot  effort.  In  the  long  run,  one  of  the 
most  harmful  beliefs  an  administrator  or  author  can  harbor  is  that  lessons  are 
written  in  a  single  sweep. 

At  one  fledgling  site,  the  staff  (entirely  composed  of  novice  authors) 
disregarded  consultants'  warnings  that  lesson  development  is  an  iterative 
process  that  should  be  interspersed  with  numerous  trial  test  runs.  The 
authors  chose  instead  to  believe  that  by  having  a  couple  of  colleagues  go 
through  their  lessons,  they  could  eliminate  virtually  all  programming  or 
design  errors. 

Within  days  of  unleashing  their  lessons  on  students,  authors  found 
themselves  frantically  fixing  errors  as  fast  as  they  were  reported  so  that 
students  could  either  get  out  of  or  get  into  various  portions  of  lessons. 
Students  were  stymied  by  questions  which  gave  them  no  clues  or  help,  and  no 
way  to  move  backward  or  forward.  Within  months  there  was  a  virtually 
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unanimous  student  mutiny,  with  nearly  all  students  asking  to  be  reassigned  to 
a  non-CBE  class. 

This  example  speaks  for  itself.  The  time  investment  in  trial  test 
runs  can  save  time  in  the  long  run.  Authors  at  this  site  felt  panicked  by  the 
fact  that  these  "fix  ups"  were  cutting  into  what  they  were  made  to  believe  was 
production  time.  Debugging  was  not  regarded  as  "producing."  A  better 
approach  would  have  been  to  schedule  trial  runs  into  the  project  time-lines 
from  the  very  beginning. 

Basically,  lesson  development  is  a  question  of  managerial  and  author 
perspective.  From  the  outset,  emphasis  should  be  placed  on  student- 
orientation.  Production  should  be  seen  as  a  multi-layered.  Iterative  process 
of  planning,  programming,  and  testing.  Inexperienced  authors  should  be 
bolstered  by  coding/design  consultants.  The  model  of  the  crash-course 
trained,  self-sufficient  author  is  as  unrealistic  as  the  concept  of  a  one-man 
expedition  to  the  North  Pole. 
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IMPLEMENTATION  AND  SITE  MANAGEMENT 

Harold  A.  Himwich 

Department  of  Research  in  Health  Education 
University  of  Connecticut  Health  Center 

In  1972,  the  Defense  Advanced  Research  Projects  Agency  (DARPA)  began  a 
large  scale  investigation  of  the  educational  and  cost  effectiveness  of  the 
PLATO  CBE  system.  This  experiment  spanned  a  period  of  almost  five  years  and, 
when  fully  implemented,  placed  more  than  100  PLATO  terminals  at  the  disposal 
of  military  users  throughout  the  country.  About  half  of  these  terminals 
were  used  either  singly  or  in  small  groups  to  explore  limited  research 
questions.  The  remainder  were  divided  among  three  military  training  sites  for 
direct  investigation  of  CBE  educational  and  cost  effectiveness.  These  sites 
shared  most  of  the  management  and  implementation  problems  of  the  smaller 
military  sites  along  with  special  difficulties  that  accompany  the  development 
of  substantial  packages  of  CBE  courseware.  In  the  smaller  sites,  for  example, 
the  limited  research  objectives  often  dictated  what  Instructional  strategies, 
if  any,  should  be  employed,  whereas  the  curriculum-development  sites  needed 
individuals  who  were  capable  of  resolving  complex  instructional  design 
problems  within  a  project's  complex  environment.  Furthermore,  with  staffs  of 
14  to  20  Individuals  at  these  sites,  managers  faced  the  problem  of  meshing  a 
variety  of  talents  into  an  effective  courseware-development  team,  while  the 
smaller  sites  often  involved  only  two  or  three  individuals.  With  their  larger 
goals  and  staff,  problems  of  management  and  implementation  and  possible 
solutions  were  somewhat  more  apparent  in  the  larger  DARPA/PLATO  sites  than  in 
the  smaller  ones.  It  will,  therefore,  be  more  informative  to  focus  attention 
on  the  richer  experiences  of  the  curriculum-development  sites. 

The  authors  at  these  sites  were  trained  by  staff  members  of  the 
Computer-based  Education  Resarch  Laboratory  (CERL)  at  the  University  of 
Illinois.  At  CERL,  the  most  striking  successes  in  using  the  PLATO  system  had 
been  obtained  by  individual  authors  who  had  possessed  whatever  subject  matter, 
programming,  and  instructional  design  expertise  they  needed  to  develop  their 
lessons.  Coming  from  this  environment,  this  authoring  model  was  naturally 


emphasized.  Moveover,  in  the  early  1970s  many  workers  at  CERL  believed 
that  it  was  easy  to  become  a  competent  PLATO  programmer.  The  managers 
of  the  large  DARPA/PLATO  sites  initially  shared  this  belief.  Consequently, 
they  dispatched  subject  matter  experts  to  CERL  for  training  in  the  happy 
expectation  that  in  a  few  weeks  time  the  wedding  of  their  subject  matter 
expertise  and  newly  acquired  PLATO  programming  conq>etence  would  produce  a 
vigorous,  new  generation  of  lesson  materials  for  their  training  programs. 

At  none  of  the  DARPA/PLATO  sites  were  these  expectations  realized.  At 
two  of  the  sites,  extensive  reorganizations  giving  the  authors  more  narrowly 
defined  roles  were  needed  to  reverse  a  trend  of  slow  and  inefficient  lesson 
deA/elopment.  At  the  third  site,  slow  development  of  lessons  of  very  uneven 
instructional  quality  compounded  with  management  problems  led  to  the 
abandonment  of  the  initial  project  objectives. 

The  unfortunate  experience  of  these  sites  with  this  particular  model 
of  the  CBE  author  reinforces  the  dictates  of  common  sense.  After  all,  a  self- 
sufficient  author  mist  be  highly  skilled  in  several  different  areas.  It  is 
unlikely,  therefore,  that  the  project  managers  would  find  in  their  staffs  an 
aggregation  of  individuals  who  uniformly  possess  the  multifaceted  talents 
needed  to  be  competent  self-sufficient  authors.  In  fact,  given  the  diversity 
in  subject  matter  knowledge,  programming  ability,  and  instructional  design 
experience  that  existed  on  these  projects,  one  would  have  least  expected  the 
self-sufficient  author  model  to  be  adopted.  To  be  sure,  part  of  the  blame 
must  be  given  to  the  initial  training  that  the  DARPA/PLATO  authors  were  given 
at  CERL.  Nevertheless,  the  inappropriate  use  of  this  model  is  currently 
prevalent  in  projects  that  did  not  have  the  same  initial  exposure  to  the  PLATO 
system.  It  thus  appears  that  some  other  factor  may  have  also  influenced  the 
adoption  of  this  model. 

The  model  of  the  self-sufficient  author  is  the  easiest  model  for 
managers  to  adopt  for  a  CBE  project  staff.  Since  a  manager  need  only  assign 
entire  CBE  lessons  to  authors,  he  can  ignore  the  complicated  problems  of  co¬ 
ordinating  the  talents  of  his  staff.  Thus,  this  model  is  often  adopted  in 
the  absence  of  any  thoughtful  plan  for  full  utilization  of  project  resources. 
As  such,  it  is  a  default  authoring  model  and  is  characteristic  of  projects 
which  lack  experienced  leadership. 

Those  DARPA/PLATO  project  managers  who  had  experience  in 
administration  and  instructional  development  did  not  give  their  authors  a  free 
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and  independent  hand  in  developing  their  lessons.  At  best,  they  adapted 
development  procedures  to  fit  the  capabilities  of  their  staffs.  While  they 
may  have  favored  a  particular  model  of  development,  the  realities  of  staff 
selection  generally  made  the  use  of  the  favored  model  impractical.  Thus,  they 
were  faced  with  assessing  their  team's  talent  at  the  project  outset  and 
adapting  lesson  development  procedures  accordingly. 

At  two  of  the  DARPA/PLATO  sites,  after  an  unsuccessful  trial  with  the 
self-sufficient  author  model,  the  project  managers  revamped  their  lesson 
development  procedures  to  conform  better  to  the  talents  of  their  staffs. 

At  one  of  these  two  sites,  there  was  little  expertise  in  instructional 
design  among  the  authors.  The  project  manager  who  had  spent  a  substantial 
portion  of  his  career  as  a  curriculum  designer  developed  a  lesson  strategy  to 
be  used  by  all  project  authors.  Thus,  the  authors  were  freed  from  design 
considerations  which  they  were  poorly  prepared  to  cope  with.  Since  the 
authors  were  mostly  weak  programmers,  the  manager  designated  the  group's 
strongest  programmer  as  the  "Senior  Author"  and  encouraged  the  other  staff 
members  to  refer  their  programming  problems  to  the  Senior  Author.  This 
procedure  not  only  freed  the  authors  from  another  task  they  were  ill-suited  to 
do  but  it  made  the  solutions  to  programming  problems  readily  available  to  the 
entire  authoring  staff  (Himwich,  1977b). 

At  the  other  site,  the  authors  were  not  subject  matter  experts  but 
were  fairly  competent  PLATO  programmers.  In  this  case,  the  project  manager, 
who  had  long  experience  with  developing  instructional  materials  for  military 
training,  brought  in  several  experienced  instructors  to  oversee  the  content  and 
presentation  of  the  lesson  materials.  In  this  project,  the  PLATO  programmers 
were  freed  from  the  problems  of  becoming  subject  matter  experts  and 
instructional  designers  to  concentrate  on  effective  use  of  the  CBE  medium. 

The  manager  of  this  project  made  the  team's  strongest  programmer  the 
programming  consultant  for  the  rest  of  the  project  staff.  As  with  the  former 
project,  a  direct  benefit  was  realized  (Himwich,  1977a,  pp.  45-53).  The 
overall  effect  of  this  reorganization  was  especially  impressive.  Whereas  650 
hours  of  authoring  time  were  needed  for  one  student  contact  hour  before  the 
reorganization  (Himwich,  1977a,  p.  40),  only  100  hours  were  needed  for  the 
same  amount  of  CBE  instruction  under  the  revision  lesson  development  scheme 
(Himwich,  1977a,  p.  49). 
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At  the  remaining  ARPA/PLATO  curriculum  development  site,  the  project 
manager  had  very  little  experience  as  an  instructional  designer*  The  only 
member  at  the  time  who  had  any  training  in  this  field  was  not  allowed  to  have 
a  significant  impact  on  lesson  development  plans  (Mlsselt,  1977,  pp.  24-25). 
The  project  manager  intended  to  treat  his  staff  "as  professionals"  and  thus 
passed  on  to  them  the  problems  of  how  to  develop  their  lessons.  This  approach 
substantially  contributed  to  the  fragmented  project  output  with  slow 
development  times,  some  highly  eccentric  and  ineffective  PLATO  lessons,  and  a 
disgruntled  staff. 

While  the  decision  of  how  to  deploy  a  staff  in  developing  CBE 
courseware  must  take  into  account  the  various  capabilities  of  the  staff 
members,  it  must  also  allow  for  the  constraints  of  project  completion  time. 

The  tighter  these  constraints,  the  more  efficient  must  be  the  production 
methods.  Such  efficiency  is  usually  obtained  by  producing  prosaic  courseware 
which  takes  little  advantage  of  the  CBE  potential  for  individualized 
instruction.  If  time  constraints  are  not  great,  it  is  possible  to  depart  from 
an  assembly  line  approach  and  assign  tasks  within  the  project  with  an  eye  to 
producing  CBE  materials  of  high  quality. 

Whatever  the  time  constraints  of  a  project,  they  must  be  considered 
carefully,  along  with  the  amount  of  courseware  to  be  produced  at  the 
outset.  The  result  of  such  an  assessment  should  be  a  time-table  for  the 
completion  of  various  project  phases.  In  addition  to  the  quantity  of 
courseware  to  be  produced,  an  estimate  of  the  number  of  authoring  hours  needed 
to  produce  a  unit  of  courseware  must  enter  into  the  determination  of  the 
project  timetable.  Although  these  timetables  and  estimates  of  production 
rates  must  be  tentative,  they  at  least  give  a  way  of  measuring  progress  during 
the  project.  Most  of  the  managers  of  the  DARPA/PLATO  sites  created  such 
timetables  and  derived  estimates  of  production  rates.  Some  of  them  erred, 
however,  in  not  using  these  estimates  to  modify  their  lesson  development 
procedures  when  early  progress  was  slower  than  anticipated.  For  most 
projects,  the  completion  date  is  not  a  variable  factor.  Thus,  when  production 
times  are  slower  than  planned,  development  procedures  must  be  modified  since 
deadline  extensions  may  not  be  possible. 
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Two  examples  from  the  ARPA/PLATO  project  illustrate  the  importance  of 
project  timetables.  At  one  of  the  sites,  lesson  development  was  based  on  the 
model  of  the  self-sufficient  author.  The  project  manager  knew  that  a 
production  rate  of  300  authoring  hours  for  each  student  contact  hour  was 
necessary  to  meet  his  objectives.  When  the  first  lessons  took  longer  than 
this  to  produce  and  when  they  were  judged  to  be  of  poor  quality  by  reviewers 
external  to  the  site,  he  reorganized  his  staff  to  steamline  production  and 
improve  lesson  quality.  In  this  case,  the  initial  timetable  helped  to  provide 
a  stimulus  for  reorganizing  the  staff  (Himwich,  1977b).  In  the  other  case,  an 
even  more  complete  timetable  was  drawn  up  also  with  an  estimate  that  300 
authoring  hours  per  student  contact  hour  would  be  needed  to  meet  the 
timetable.  This  estimate  had  failed  to  account  for  non-authoring  duties  as 
well  as  several  other  time-consuming,  but  forseeable,  problems  that  the 
project  suffered  in  its  early  stages  (Misselt,  1977,  pp. 22-25).  The  initial 
authoring  model  employed  at  this  site,  actually  adopted  by  default,  was  that 
of  the  self-sufficient  author.  As  time  pressures  mounted  for  this  project, 
the  manager  made  no  attempt  to  modify  lesson  development  procedures  although 
such  modification  was  clearly  indicated. 

These  two  cases  not  only  point  up  the  importance  of  using  one's 
timetable  once  it  has  been  derived,  but  also  the  damage  that  authors 
attempting  to  be  self-sufficient  can  do  to  such  timetables.  Part  of  this 
damage  originates  from  the  fact  that  most  authors  are  not  skilled  enough  to  be 
self-sufficient  so  that  they  tend  to  work  very  slowly  as  they  acquire  needed 
skills.  However,  the  main  problem  arises  from  the  fact  that  authors  working 
in  this  manner  tend  to  view  a  CBE  lesson  as  an  Infinitely  perfectable  work. 
Being  naturally  sensitive  to  the  shortcomings  of  their  work,  they  continue  to 
polish  them  unless  their  access  to  the  lessons  is  actually  restricted. 

While  the  most  excessive  and  wasteful  expenditures  of  time  and  labor 
occur  when  a  lesson  has  a  single  author  and  when  that  author  has  complete 
automony  in  determining  when  his  work  is  complete,  a  similar  sort  of  problem 
can  arise  independent  of  the  lesson-production  procedures  being  used. 

Although  this  problem  may  take  on  different  appearances,  it  always  arises  from 
the  fact  that  any  CBE  lesson  can  be  improved.  Thus,  a  lesson-production  unit, 
whether  it  is  a  team  or  a  single  individual,  can  always  find  ways  to  work  on 
satisfactory  lessons.  For  this  reason,  it  Is  essential  that  project  managers 
devise  criteria  for  determining  lesson  completion  and  apply  them  vigorously. 
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In  summary,  experience  with  the  ARPA/PLATO  project  suggests  that  the 
following  management  precepts  are  vital  to  the  success  of  any  CBE  project: 

1.  The  model  of  the  author  as  a  self-sufficient  combination  of 
programming.  Instructional  design,  and  subject  matter 
expertise  is  not  realistic  in  most  CBE  production  environments. 

2.  A  division  of  labor  in  lesson  development  should  therefore 
be  established  to  take  advantage  of  available  resources. 

3.  A  list  of  project  milestones  and  a  timetable  for  their 
attainment  should  be  drawn  up  early  in  the  project  so 
that  estimates  of  progress  can  be  made  and  appropriate 
remedial  actions  taken  when  needed. 

4.  The  determination  of  when  a  portion  of  CBE  courseware  is 
completed  should  not  reside  in  the  hands  of  the  people 
directly  involved  with  the  courseware. 

The  apparent  a  priori  validity  of  these  ideas  would  lead  one  to  believe  that 
they  are  nothing  more  than  common  sense  applied  to  CBE  development.  During 
the  DARPA/PLATO  project,  however,  they  were  frequently  ignored.  For  this 
reason  and  the  fact  that  a  survey  of  current  CBE  projects  shows  only  haphazard 
application  of  these  ideas,  it  can  only  benefit  us  to  be  reminded  of  them. 
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RESULTS  AND  FINAL  EVALUATIONS 

R.  A.  Avner 

University  of  Illinois  at  Urbana-Champaign 

The  mature  computer-based  education  (CBE)  production  effort 
constitutes  an  end  point  of  both  a  product  and  a  process.  By  evolution  or  by 
simple  trial-and-error,  mature  production  groups  are  most  likely  to  have 
developed  both  exemplary  applications  of  the  medium  and  exemplary  procedures 
for  efficient  production  of  such  applications.  Even  where  these  goals  are  not 
met,  the  fact  that  a  group  has  gone  through  the  full  production  cycle  one  or 
more  times  means  that  there  are  probably  valuable  lessons  to  be  learned  from  an 
examination  of  its  history. 

The  findings  reported  here  are  based  on  longitudinal  studies  of  22 
instructional  design  groups,  cross-sectional  studies  of  six  additional  groups, 
and  observations  made  during  more  than  100  evaluations  of  major  CBE  design 
projects.  Consistent  findings  from  these  studies  over  the  past  18  years 
suggest  that  it  is  now  possible  to  make  several  general  statements  about  the 
relationships  among  group  productivity,  group  structure,  experience,  quality 
of  work,  and  work  environment. 

PROCESS  CHARACTERISTICS 

Mature  CBE  production  groups  show  patterns  of  both  capabilities  and 
working  methods  which  are  distinct  from  those  of  beginning  groups.  In  some 
cases  the  differences  are  only  cosmetic,  involving  little  more  than  facility 
with  the  jargon  and  the  nythology  of  the  medium.  In  other  cases  the 
differences  are  more  substantial  and  reveal  themselves  in  highly  efficient 
production  of  effective  Instructional  material.  The  most  consistent 
difference  seen  between  beginning  and  mature  groups  is,  as  would  be  expected, 
a  higher  rate  of  productivity.  However  this  difference  is  only  partially  the 
result  of  higher  levels  of  experience  among  individuals  within  mature  groups. 
Even  in  groups  with  relatively  high  rates  of  personnel  turnover,  the 
established  organization  can  show  efficiencies  as  a  result  of  knowing  what  is 
essential  and  what  is  not  essential  in  the  training  of  new  workers. 

Established  groups  also  show  heavy  use  of  standardized  repertoires  of 
procedures  which  eliminate  time-consuming  exploratory  efforts. 
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Differences  between  mature  groups  with  equivalent  amounts  of  exposure 
to  CBE  seem  related  to  three  major  factors:  (1)  the  extent  and  pattern  of 
production  demands  placed  on  the  group,  (2)  the  variety  of  Instructional 
settings  In  which  a  group  produced  materials,  and  (3)  the  degree  of  commitment 
toward  producing  measurable  changes  In  student  performance* 

Production  demand*  Studies  of  Individual  author  productivity  (Avner, 
1979)  show  that  one  of  the  major  factors  determining  production  rate  Is  the 
extent  to  which  authors  experience  a  continued  pressure  of  short-term 
deadlines  for  production  of  material*  Authors  who  have  met  months  or  years  of 
two-  to  three-week  deadlines  for  producing  completed  CBE  Instructional  modules 
show  production  rates  that  are  a  several  times  higher  than  those  of 
individuals  who  have  never  met  continual  short-term  deadlines*  Production 
rates  of  30  author  hours  per  hour  of  student  material  are  not  uncommon  even  in 
Instances  where  pedagogical  structure  as  well  as  content  must  be  produced* 

Variety  of  experience*  The  quality  of  material  produced  when  a  group 
is  faced  with  changes  in  student  population  or  changes  in  types  of  instruction 
required  seems  highly  related  to  the  range  of  past  experience  of  a  group* 
Groups  which  have  extensive  experience  In  providing  instruction  of  a  single 
type  to  a  restricted  population  of  students  tend  to  overgeneralize  procedures 
which  were  found  to  be  effective  in  that  situation*  This  tendency  toward 
overgeneralization  is  particularly  virulent  when  the  past  experience  was  very 
successful* 

Performance  orientation*  Development  of  effective  instructional 
approaches  by  a  group  seems  ultimately  to  depend  on  a  commitment  of  the  group 
or  its  leadership  to  production  of  material  of  demonstrable  effectiveness*  No 
group  is  ever  likely  to  admit  a  desire  to  produce  poor  quality  material,  but 
many  groups  demonstrate  a  head-in-the-sand  approach  to  production  which  almost 
assures  such  an  outcome*  Groups  where  frequent,  early  testing  with  students 
is  used  to  gather  data  which  are  immediately  used  in  a  revision  process  show  a 
high  likelihood  of  producing  both  high  quality  material  and  a  repertoire  of 
general  procedures  that  will  assure  efficient  future  production  of  similar 
materials*  Groups  which  concentrate  only  on  measures  of  quantity  of  output 
show  a  high  likelihood  of  producing  both  low  quality  material  and  a  repertoire 
of  procedures  for  efficient  production  of  similar  low-quality  materials* 
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Unfortunately,  quantity  Is  easier  to  measure  than  quality,  and  far  too  many 
groups  gradually  yield  to  use  of  module  or  lesson  counts  as  the  sole  index  of 
group  effectiveness. 

Administrative  considerations  also  differ  in  type  and  importance  for 
new  and  mature  CBE  projects.  The  mature  production  effort  is  faced  with  a 
class  of  problems  that  the  beginning  group  sees  In  only  a  limited  fashion. 
Foremost  among  these  problems  are  those  of  maintenance,  documentation  ,  and 
updating  of  existing  materials.  Some  successfully  implemented  materials  have 
died  an  agonized  death  because  of  insufficient  planning  for  long-range 
support.  With  turnover  of  personnel,  inadequate  documentation  frequently 
proves  the  undoing  of  projects  which  have  to  devote  more  time  to  upkeep  of 
packages  than  was  invested  in  the  original  production. 

Confusion  between  start-up  and  steady-state  costs  and  the  fact  that 
these  costs  continually  change  in  relative  importance  during  development  of  a 
project  causes  administrative  headaches  to  projects  at  all  stages  of  life. 

New  groups  may  either  ignore  start-up  costs  or  mistake  them  for  steady-state 
costs.  Mature  groups  are  well  aware  of  the  burden  of  steady-state  maintenance 
costs  but  may  erroneously  assume  that  they  have  left  major  start-up  costs 
behind  them  forever. 

PRODUCT  CHARACTERISTICS 

In  a  field  such  as  CBE  where  major  learning  is  still  going  on  among 
experienced  practitioners,  groups  which  communicate  effective  techniques  to 
other  groups  can  be  considered  to  have  produced  a  product  that  is  at  least  as 
valuable  as  the  instructional  materials  designed  for  more  typical  students. 

In  fact,  several  of  the  groups  studied  had  research  rather  than  production 
goals  and  intended  to  produce  reports  which  would  communicate  effective 
techniques  for  use  of  CBE. 

Formal  reports  of  instructional  development  efforts  by  mature  groups 
were  rarely  in  a  form  which  would  convince  others  of  the  efficacy  of  the 
medium  and  only  slightly  more  frequently  in  a  form  which  showed  convincing 
evidence  of  the  effectiveness  of  the  design  effort  itself.  This  paucity  of 
evaluative  output  was  particularly  evident  in  academic  settings  and  in 
projects  operating  on  a  job-shop  basis.  Single-task  oriented  groups 
(especially  those  with  research  goals)  showed  a  higher  Incidence  of  evaluative 
efforts  in  final  product  descriptions;  however  even  these  were  often  limited 


to  provision  of  evidence  that  efforts  had  been  expended  and  products 
generated. 

Attempts  to  communicate  CBE  instructional  design  skills  to  others  by 
journal  articles  or  technical  reports  often  seemed  inversely  related  to  the 
actual  record  of  productivity  of  a  group.  This  relationship  seems  most  likely 
the  result  of  the  fact  that  such  reports  are  often  motivated  by  professional 
or  contractual  demands  which  are  rarely  present  in  job-shop  production  (the 
setting  where  the  greatest  depth  and  range  of  experience  is  likely  to  exist). 
In  several  instances  it  was  found  that  groups  said  less  and  less  publicly 
about  methods  of  CBE  design  as  their  experience  grew.  As  a  result,  the 
several  excellent  publications  and  planning  guides  available  for  those  seeking 
initial  aid  in  structuring  design  efforts  are  not  supported  by  an  equivalent 
literature  giving  support  for  the  practical  problems  of  efficient  production 
of  effective  material. 

Both  the  disappointing  lack  of  formal  evidence  for  effectiveness  of 
materials  and  the  lack  of  public  descriptions  of  effective  CBE  applications 
and  techniques  are  probably  a  predictable  outcome  of  the  implicit  goals  of 
most  of  the  projects  observed.  Production  effort’s  rarely  had  to  validate 
their  materials  in  a  form  designed  to  convince  persons  other  than  local 
consumers.  Research  efforts  often  focused  on  methods  designed  ,to  work  with  a 
wide  variety  of  media,  with  the  result  that  techniques  unique  to  CBE  were 
sometimes  actually  avoided.  Despite  these  disappointing  trends,  effective 
materials  and  techniques  were  developed  and  were  documented  in  some  cases 
(Avner,  1978). 

OUTCOMES 

The  probability  of  survival  of  the  mature  projects  observed  seemed  to 
be  related  more  to  their  reason  for  establishment  than  to  the  quality  and 
quantity  of  their  work.  Projects  organized  for  single  tasks  (e.g.  to  carry 
out  a  contract  or  produce  a  course  in  a  given  area)  showed  the  highest 
probability  of  being  dissolved  after  completion  of  the  task.  Given  the  stated 
goals  of  such  a  project,  this  outcome  is  certainly  to  be  expected. 
Nevertheless,  in  most  cases  it  turned  out  to  be  an  outcome  unintended  by 
members  of  the  projects.  With  only  rare  exceptions,  organized  groups 
attempted  to  extend  their  existence  beyond  the  completion  of  the  intitial  goal 
by  seeking  funding  for  additional  projects.  Based  on  the  evidence  for 
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in^roved  productivity  by  these  mature  groups,  such  efforts  are  certainly  in  the 
best  interests  of  the  educational  community.  Unfortunately,  where  added 
funding  was  not  immediately  forthcoming,  experienced  members  of  such  groups 
were  usually  quickly  lured  away  by  other  production  groups,  and  the  original 
group  was  left  with  insufficient  resources  to  support  further  production 
efforts  even  if  the  funding  became  available. 

In  conclusion,  mature  CBE  design  groups  that  have  produced  a  wide 
variety  of  materials  under  continuous  moderate  deadline  pressure  possess 
skills  in  quality  and  quantity  that  go  well  beyond  those  possessed  by 
beginning  groups.  These  skills  have  the  potential  for  allowing  production  of 
instructional  products  of  superior  effectiveness.  However,  information  about 
the  nature  of  these  skills  or  the  effectiveness  of  resulting  products  is 
seldom  being  communicated  well  by  members  of  experienced  groups.  Finally, 
continued  survival  of  groups  is  rarely  related  to  their  production  record. 
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APPENDIX  I 


ANNOTATED  LIST  OF  MTC  REPORTS 


These  reports  contain  the  detailed  information  that  forms  the  basis 
for  this  final  report  as  well  as  providing  an  historical  commentary  on  the 
evolutionary  growth  of  experience  of  the  application  of  CBE  in  military 
settings. 

Except  'Aiere  noted,  reports  in  this  series  are  available  at  cost  from 
the  Documentation  Office  of  the  Computer-Based  Education  Research  Laboratory, 
University  of  Illinois,  Urbana  IL  61801.  ERIC  accession  numbers  are  also 
given  for  reports  available  from  Educational  Research  Information  Centers. 

MTC  Paper  1  (August  21,  1972)  D.  V.  Meller 

Terminal  Distribution  and  Use.  (3  pp)  (Internal  use  only) 

This  early  report  gives  preliminary  estimates  of  use  of  CBE  terminals 
during  the  development  and  initial  evaluation  of  instructional  materials.  It 
was  used  as  an  aid  to  earliest  planning  for  placement  of  terminals  at  military 
sites  before  direct  experience  was  available. 

MTC  Paper  2  (October  26,  1972)  D.  V.  Meller 
Reviewing  a  PLATO  Lesson.  (11  pp)  (Internal  use  only) 

An  outline  and  checklist  of  procedures  intended  to  increase  the 
likelihood  that  valuable  student  test  time  would  not  be  wasted  in  detection  of 
errors  and  problems  that  could  more  economically  be  detected  by  the  designer 
or  her/his  peers. 

MTC  Report  3  (October,  1975)  Larry  Francis,  Merle  Goldstein,  &  Eileen 
Call-Himwich 

Lesson  Review.  (154  pp)  (ED  124  132) 

A  guide  to  techniques  and  uses  of  reviews  of  CBE  lessons,  including 
step-by-step  procedures,  samples,  and  practice  reviews.  Based  on  techniques 
experimentally  demonstrated  to  be  effective  in  improving  the  usefulness  of 
review  comments  by  persons  with  instructional  design  experience,  this  report 
covers  practical  problems  such  as  selection  of  reviewers,  the  sociology  of 
reviewing,  and  potential  negative  outccmes  of  reviewing. 
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MTC  Report  4  (July,  1974)  J.  P.  Neal 

The  CGE-PLATO  Electronic  Laboratory  Station  Structure  and  Operation.  (99  pp) 
(ED124145) 

This  report  describes  an  application  of  CBE  in  simulations  used  in 
training  for  the  use  of  particular  items  of  equipment-  Technical  details  of 
interface  design  and  logic  are  presented  for  a  training  application  which 
requires  the  computer  to  sense  interconnections  between  elements  of  electronic 
circuitry  and  test  equipment  as  well  as  panel  settings  of  the  electronic  test 
equipment . 

MTC  Report  5  (December,  1974)  J.  P.  Neal 

Electronic  Laboratory  Instruction  Using  the  CGE-PLATO  Laboratory  Station. 

(Ill  pp)  (available  from  CERL  only) 

A  brief  summary  of  the  philosophy  and  technique  of  using  the  Computer- 
Guided  -Experimentation  approach  in  instruction  is  presented  in  this  report 
together  with  a  review  of  opinions  and  usage  of  students  in  a  course  on 
introductory  electronic  laboratory  techniques  which  used  CGE-PLATO. 

MTC  Report  6  (December,  1974)  J.  P.  Neal 

The  CGE-PLATO  Electronic  Laboratory  Instructional  Programs.  (232  pp) 

(ED  124  146) 

This  report  gives  source  code  listings  in  the  TUTOR  language  for  12 
instructional  lessons  designed  for  use  with  the  CGE-PLATO  equipment. 

MTC  Report  7  (November,  1976)  Larry  Francis 

The  TUTOR  Training  Course;  Lessons  Learned.  (126  pp,  7  Appendixes) 

(ED  135  377) 

Approximately  100  new  authors  were  taught  to  use  the  TUTOR  language 
over  a  period  of  three  years.  The  two-  to  three-week  sequence  developed  by 
the  MTC  Group  was  the  first  formal  author  training  program  for  the  TUTOR 
language.  The  materials  were  developed  in  an  evolutionary  fashion  to  meet 
needs  of  new  CBE  authors  working  in  military  settings.  The  report  details 
major  problems  encountered,  solutions  applied,  and  evaluation  of  the  final 
form  of  the  instruction.  Findings  and  conclusions  appropriate  to  the  needs  of 
instructors  of  new  CBE  authors,  developers  of  author  training  materials,  and 
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managers  of  CBE  development  centers  are  highlighted.  Included  are  13 
principles  found  effective  in  teaching  this  material,  four  unresolved  problems 
encountered,  examples  of  good  and  poor  techniques,  standards,  and  expected 
completion  times  for  the  sequence. 

MTC  Report  8  (January,  1977)  Eileen  Call-Himwlch 

An  Assessment  of  Lesson  Review  as  a  Formative  Evaluation  Tool.  (33  pp) 

(ED  140  775) 

Lesson  reviews  can  potentially  provide  both  a  means  of  increasing 
lesson  quality  and  a  means  of  increasing  the  design  skills  of  lesson  authors. 
This  report  describes  the  evolution  of  the  lesson  review  process  developed  by 
the  MTC  Group  over  a  three-year  period.  Aspects  of  the  process  found  to  be 
crucial  to  effectiveness  included  reviewer  location  (on-site  reviews  were  more 
effective  than  those  done  at  a  distance),  reviewer  status  (reviewers  with 
perceived  authority  equal  to  that  of  the  designer  were  more  effective  than 
those  with  higher  or  lower  status),  review  format  (rapid  reviews  of  work  in 
progress  on  points  mutually  agreed  to  by  both  reviewer  and  reviewee  had  more 
impact  on  the  final  product  than  reviews  done  on  completed  products  and  based 
only  on  the  reviewer's  perception  of  needs). 

MTC  Report  9  (December,  1976)  Larry  Francis 
PLATO  IV  Terminal  Peripheral  Devices.  (69  pp) 

(ED  135  378) 

Empirical  results  of  implementations  of  three  major  PLATO  IV 
peripheral  devices  (the  microfiche  system,  the  touch  panel,  and  the  random- 
access  audio  device)  are  examined.  In  addition  to  listing  operating 
characteristics  of  these  devices,  an  assessment  of  efforts  and  skills  required 
for  their  use  and  an  examination  of  alternatives  was  made.  Intended  mainly  as 
an  aid  to  decision  makers,  the  report  emphasizes  managerial  considerations  In 
use  of  these  devices.  It  notes  specific  drawbacks  and  concludes  that  on-site 
testing  and  maintenance  are  needed  for  reliable  performance.  Although  some 
hardware  considerations  are  discussed,  the  basic  approach  is  from  a  human- 
factors  viewpoint. 


35 


MTC  Report  10  (January,  1977)  Joseph  A.  Klecka 
An  Overview  of  Chanute  Lessons.  (80  pp) 

(ED  151  015) 

A  survey  of  current  lesson  development  was  considered  essential  to 
determine  the  quality  and  effectiveness  of  instructional  material  produced  on 
the  PLATO  IV  computer-assisted  instruction  system.  The  trial  period  lasted 
for  several  years.  Both  civilian  and  military  personnel  developed  the 
lessons,  in  consultation  with  the  Military  Training  Centers  Group  at  the 
Computer-Based  Education  Research  Laboratory,  University  of  Illinois,  for  the 
Defense  Advanced  Research  Projects  Agency.  Eight  representative  lessons  were 
selected  by  a  Chanute  staff  member.  Then  an  evaluator  prepared  in-depth 
reviews  of  each,  both  in  the  formative  and  summative  stages.  Data  gathered 
from  the  individual  analyses  formed  the  basis  for  the  final  report.  It 
indicated  general  trends  and  Isolated  problem  areas  in  the  instructional 
design.  The  study  resulted  in  an  assessment  of  the  instructional 
effectiveness  and  utilization  of  the  pedagogical  capabilities  of  the  PLATO 
system  in  one  training  environment. 

MTC  Report  11  (March,  1977)  Joseph  A.  Klecka 

Three  Aspects  of  PLATO  Use  at  Chanute  AFB;  CBE  Production  Techniques.  Computer- 
Aided  Management,  and  Formative  Development  of  CBE  Lessons.  (79  pp) 

(ED  151  015) 

Lesson  production  was  studied  during  the  process  of  instructional 
system  development  (ISD)  at  Chanute  AFB.  This  report  considers  four  major 
factors  influencing  lesson  production;  implementation  of  the  "lean  approach", 
the  ISD  role  in  lesson  production,  the  transfer  of  programmed  instruction 
techniques  to  CBE,  and  the  general  method  of  lesson  production  (group  vs. 
individual).  Each  factor  is  discussed  in  terms  of  its  effect  on  the  quantity 
and  quality  of  lesson  production. 
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MTC  Report  12  (June,  1977)  Esther  R.  Steinberg  (Editor),  with  R.  A.  Avner,  Eileen 
Call-Hlmwlch,  Larry  Francis,  H.  A.  Himwlch,  Joseph  A.  Klecka,  A.  Lynn  Mlsselt, 
and  Esther  R.  Steinberg 

Critical  Incidents  In  the  Evolution  of  PLATO  Prolects.  (70  pp) 

(ED  148  298) 

This  report  Is  Intended  to  serve  as  a  resource  for  the  development  of 
management  and  Instructional  guidelines  for  computer-based  education  (CBE), 
Although  the  data  In  It  were  gathered  from  PLATO  projects  only,  they  represent 
projects  which  varied  widely  In  target  populations  (elementary  through 
professional  students),  subject  matter  content,  type  of  Implementation,  and 
size  and  scope.  Therefore,  the  report  provides  Information  useful  both  to 
PLATO  users  and  to  developers  of  CBE  In  general. 

Critical  incidents  are  defined  In  terms  of  four  criteria,  then  more 
than  125  case  histories  of  critical  incidents  are  documented.  These  histories 
are  organized  by  topics,  rather  than  projects,  so  as  to  provide  a 
taxonomy  of  matters  or  Issues  which  are  critical  during  project  development. 

The  report  also  Includes  summaries  and  analyses  of  the  processes  and 
procedures  and  their  subsequent  effects. 

MTC  Report  13  (August,  1977)  Larry  Francis 

A  Comparison  of  the  Costs  for  Illustrations  Presented  by  the  Plasma  Panel  and 
Microfiche .  (17  pp) 

(ED  148  299) 

The  facilities  of  the  microfiche  projector  and  the  plasma  panel 
overlap  somewhat.  Each  has  its  unique  capabilities  and  constraints,  but  in 
many  cases  either  can  be  used.  Some  authors  and  site  directors  consistently 
choose  microfiche  over  plasma  displays,  or  vice  versa,  because  they  are 
convinced  one  Is  considerably  less  expensive.  However,  to  the  best  of  our 
knowledge,  no  cost  comparisons  have  been  published.  This  study  was  undertaken 
to  explore  the  costs  associated  with  each  display  technique  for  those  cases 
where  either  suffices. 
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MTC  Report  14  (September,  1977)  H.  A.  Himwlch  (Editor),  with  A.  Lynn  Misselt, 
H.  A.  Himwich,  Larry  Francis,  R.  A.  Avner,  Kikumi  Tatsuoka,  and  Joseph  Klecka 
Critique  and  Summary  of  the  Chanute  AFB  CBE  Project.  (144  pp) 

(ED  150  947) 

This  report  consists  of  two  parts:  a  site  history  and  summary  and  a 
critique  and  expansion  of  an  Air  Force-produced  evaluation  of  the  Chanute 
PLATO  project.  The  site  history  traces  the  project  through  a  variety  of 
management  and  lesson  development  styles.  These  supplement  the  history 
described  briefly  in  the  Air  Force  report. 

The  critique  portion  of  this  report  analyzes  the  Air  Force  report 
chapter  by  chapter.  For  each  section  comments,  elaborations,  or  further 
statistical  treatments  are  provided.  The  final  comment  in  this  report  agrees 
with  a  basic  conclusion  of  the  Air  Force  report  by  stating,  "we  suggest  that 
readers  of  the  AFHRL  report  not  view  the  Chanute  experience  as  an  example  of 
how  training  can  be  structured  around  a  CBE  system  to  take  maximum  advantage 
of  its  capabilities,  but  as  an  example  of  how  PLATO  can  be  Incorporated  into  a 
traditional  military  training  environment  without  causing  changes  in  basic 
routines ." 

MTC  Report  15  (September,  1977)  Larry  Francis  and  Tamar  Weaver 
Analysis  of  Student  Interaction  Data  from  CBE  Lessons.  (26  pp) 

(ED  152  237) 

Features  of  the  PLATO  IV  CBE  system  allow  the  gathering  of  large 
quantities  and  varieties  of  Information  describing  the  interactions  of 
students  with  CBE  courseware.  Until  recently  this  type  and  volume  of  data 
have  not  been  easily  available,  and  hence  few  techniques  or  guidelines  for  its 
analysis  have  been  investigated.  This  report  describes  some  initial  attempts 
and  the  current  status  of  efforts  to  collect,  condense,  and  analyze  these  data 
for  the  purpose  of  diagnosing  student  problems  and  improving  instruction. 
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MTC  Report  16  (October,  1977)  H.  A.  Hlmwich 

A  Comparison  of  the  TICCIT  and  PLATO  System  In  a  Military  Setting.  (26  pp) 

(ED  152  238) 

This  report  describes  the  initial  phases  of  a  pilot  test  to  determine 
the  effectiveness  of  computer-based  network  systems  to  teach  geographically 
dispersed  students  enrolled  in  Air  Force  extension  courses  at  the  Air 
University.  The  project  became  a  comparison  between  the  PLATO  and  TICCIT 
systems.  As  far  as  we  know,  this  project  is  the  only  one  in  which  the  two 
systems  were  compared  side-by-side,  developing  materials  for  the  same 
objectives,  and  being  staffed  by  comparably  qualified  people. 

MTC  Report  17  (October,  1977)  Larry  Francis 

Guidelines  for  Establishing  and  Managing  a  Computer-Based  Education  Site. 

(140  pp)  (ED  150  966) 

Management  guidelines  are  presented  for  three  crucial  phases.  Part  I 
contains  a  number  of  questions  which  site  staff  members  should  try  to  answer 
when  establishing  or  while  operating  a  computer-based  education  (CBE)  site. 
Each  question  is  followed  by  a  brief  commentary.  Part  II  contains  suggestions 
for  selecting  and  training  the  staff  of  a  site.  In  order  to  keep  the 
commentary  to  the  questions  in  part  I  brief,  detailed  discussions  of  some 
topics  have  been  placed  in  part  III  of  these  guidelines.  Part  III  also 
contains  recommendations  and  suggestions  about  topics  not  Introduced  in  parts 
I  and  II. 

MTC  Report  18  (November,  1977)  Eileen  Call-Himwlch  and  Esther  R.  Steinberg 
Myth  and  Reality;  Essential  Decisions  in  Computer-Based  Instructional  Design. 
(38  pp)  (ED  152  239) 

Most  "how-to"  guides  start  from  the  beginning  and  work  forward.  This 
one  works  backward.  It  starts  with  a  number  of  common  CAI  author  "myths" 
(misconceptions)  and  their  unpleasant  consequences,  then  works  backward  to 
discuss  how  the  matter  might  better  have  been  approached  in  the  first  place. 
For  this  reason,  this  guide  is  not  so  much  an  attempt  to  Instruct  as  to 
forewarn. 


39 


MTC  Report  19  (December,  1977)  H.  A.  Hlmwich  (Editor),  with  H.  A.  Himwich, 
Eileen  Call-Himwich,  R.  A.  Avner,  and  A.  Lynn  Misselt 
Summary  and  Analysis  of  the  Aberdeen  CBE  Project.  (144  pp) 

The  U.  S.  Army  Ordnance  Center  and  School  studied  the  acceptability  of 
CBE  instruction  to  students  and  instructors,  the  reliability  of  the  PLATO  CBE 
system,  its  cost  and  instructional  effectiveness,  and  resources  needed  for 
preparation  of  effective  CBE  materials.  The  implementation  was  Judged  to  be 
generally  successful,  and  PLATO  was  found  to  be  acceptable  on  all  points  except 
cost  for  the  student  population  involved.  Products  of  the  instructional 
design  approach  used  by  this  project  were  assessed  and  Judged  to  be  successful 
in  meeting  short-term  Instructional  goals.  Pilot  studies  compared  mastery¬ 
learning  materials  Implemented  on  CBE  to  parallel  materials  implemented  on 
other  individualized  media.  The  CBE  materials  gave  equivalent  learning  with 
significantly  greater  time  savings  than  alternative  individualized  media. 

MTC  Report  20  (January,  1978)  Larry  Francis,  Joseph  A.  Klecka,  and  A.  Lynn 
Misselt 

Selected  Characteristics  of  Tutor  Programs  Produced  in  ARPA  Sponsored  PLATO 
Projects.  (31  pp) 

(ED  151  018) 

The  purposes  of  the  study  were: 

1.  To  demonstrate  the  potential  of  computer-based  scanning  of  CBE  programs 
as  a  means  for  extracting  Information  about  the  programming  techniques  used. 

2.  To  suggest  the  use  of  computer-scanning  techniques  to  aid  management  of 
CBE  development  efforts. 

3.  To  provide  additional  information  about  the  products  of  CBE  lesson 
development  projects  conducted  under  DARPA  sponsorship. 

The  scope  of  the  study  was  limited  to  include  only  a  few  of  the  more 
interesting  and  potentially  useful  parameters  that  can  be  easily  measured  by 
computer-scanning  techniques.  The  lesson  characteristics  selected  for 
examination  were  chosen  on  the  basis  of  (a)  the  ease  and  reliability  of  their 
measurements,  (b)  their  Importance  in  CBE,  and,  (c)  their  usefulness  in 
illustrating  the  potential  of  this  technique. 


MTC  Report  21  (January,  1978)  A.  Lynn  Mlsselt  and  Eileen  Call-Himwlch 
Analysis  of  Sheppard  AFB  Computer-Based  Education  Project.  (100  pp) 

(ED  160  107) 

This  report  summarizes  two  aspects  of  the  "Problem  Oriented  Medical 
Curriculum"  developed  as  part  of  the  Sheppard  Air  Force  Base  PLATO  project. 

The  first  part  of  the  report  summarizes  and  discusses  the  project  history. 
Project  outcomes  are  analyzed  and  an  overall  summary  of  the  project  is 
provided. 

In  the  second  part  of  the  report  the  Sheppard  cousevare  is  examined. 
Included  are  discussions  of  problem-orientation,  lesson  design,  instructional 
approaches,  and  lesson  development. 

MTC  Report  22  (January,  1978)  Kikumi  Tatsuoka 

Approaches  to  Validation  of  Criterion-Referenced  Tests  and  Computer-Based 
Instruction  in  a  Military  Project.  (60  pp) 

(ED  161  434) 

Problems  Involved  in  setting  validation  criteria  for  mastery-learning 
of  CBE  lesson  materials  are  examined  in  this  report.  Several  models  of 
performance  scores  were  tested  against  actual  data  from  materials  developed 
at  a  DARPA  CBE  site.  The  Bayesian  binomial  model  (beta  binomial)  matched 
data  better  than  a  sample  binomial  model  and  is  recommended  for  setting 
lesson  validation  criterion  levels  where  the  assumptions  for  this  model  are  met 

MTC  Report  23  (January,  1978)  Kikumi  Tatsuoka  (Editor),  with  Kikumi  Tatsuoka, 

A.  Lynn  Mlsselt,  Joseph  Klecka,  and  Patrick  Marltz 

Attitude  and  Performance  of  Military  Students  and  Instructor  Attitude  in 
Computer-Based  Technical  Training.  (61  pp) 

Attitudes  of  students  and  instructors  were  measured  by  several 
Instruments.  Factor  analytic  and  multiple  regression  techniques  were 
employed  in  assessing  student  attitude  and  performance  measures.  It  was 
found  that  mechanical  and  system  failures  had  no  significant  effect  on  student 
performance  but  did  have  a  negative  effect  on  their  attitude  toward  use  of 
CBE.  Students  using  CBE  materials  showed  stable,  reliable  positive  attitudes 
toward  the  program  while  those  not  using  CBE  showed  attitudes  which  varied 
with  the  popularity  of  their  instructors. 


F 


Instructor  attitudes  were  generally  positive  toward  the  CBE  program 
and  materials  but  showed  negative  effects  after  experiencing  failures  of  CBE 
system  service.  Student  attitudes  toward  the  CBE  program  were  seen  as  being 
affected  by  instructor  attitudes. 

MTC  Report  24  (September,  1980)  Allen  Avner 
Measuring  and  Reporting  CBE  System  Reliability.  (35  pp) 

Measures  of  computer  system  reliability  commonly  used  for  batch  and 
interactive  computer  applications  are  not  necessarily  appropriate  to  the  needs 
of  CBE  systems.  This  report  reviews  the  major  factors  involved  in  determining 
the  effect  of  unreliability  on  system  availability  for  CBE.  Results  of 
an  empirical  study  of  the  relationship  between  various  indexes  of  overall 
system  availability  and  user  attitudes  are  described.  One  index  (time- 
weighted-problems-per-unit-tlme)  was  found  to  be  highly  correlated  with 
user  attitude  and  to  have  properties  (such  as  additivity)  which  make  it 
very  convenient  for  measuring  contributions  of  component  unreliability  to 
total  system  reliability. 
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